Brain activity measurement and feedback system

ABSTRACT

A head set (2) comprises a brain electrical activity (EEG) sensing device (3) comprising EEG sensors (22) configured to be mounted on a head of a wearer so as to position the EEG sensors (22) at selected positions of interest over the wearers scalp, the EEG sensing device comprising a sensor support (4) and a flexible circuit (6) assembled to the sensor support. The sensor support and flexible circuit comprise a central stem (4a, 6a) configured to extend along a center plane of the top of the head in a direction from a nose to a centre of the back of a wearers head, a front lateral branch (4b, 6b) configured to extend across a front portion of a wearer&#39;s head extending laterally from the central stem, a center lateral branch (4c, 6c) configured to extend across a top portion of a wearer&#39;s head essentially between the wearer&#39;s ears, and a rear lateral branch (4d, 6d) configured to extend across a back portion of a wearer&#39;s head. The sensor support (4) comprises a base wall (401) and side walls (402) extending along edges of the base wall to form an essentially flat “U” shaped channel (403) in which the flexible circuit (6) is inserted and the base wall comprise EEG sensor orifices (404) to allow access to the EEG sensor contacts or electrodes on the flexible circuit.

TECHNICAL FIELD

The present invention relates to a head-mounted device to measure ormonitor cortical activity of a user, and optionally to generate stimuliand/or to provide feedback to the user. Applications for use of theinvention may include gaming, training, learning of sensory-motorskills, diagnosis or treatment of neurological injury or disease.

DESCRIPTION OF RELATED ART

Neurological injury which follows a stroke often manifests ashemiparesis or other partial paralysis of the body. Currentrehabilitation procedures are often based on exercises performed by theimpaired body part, the movement of which is tracked in real-time toprovide feedback to the patient and/or a medical practitioner. Computercontrolled mechanical actuation systems have been used to track aposition of, and force applied by, a body part such an arm of a patientas a predetermined movement pattern is executed by the patient. Toreduce patient fatigue such systems can support the patient, for exampleby actuators which can assist during execution of the movement. Adisadvantage of such devices is that they can be complicated andexpensive. Also, conventional systems are based on tracking actualmovements and are therefore not adapted for diagnosis or treatment inthe very early stages after an occurrence of stroke where movement isimpaired or very limited. They may also present a risk to the patent if,for example, the body part is moved too quickly or if part of the heavyactuation equipment falls on the patent. They are also not particularlyportable, which generally prohibits home use and use in a hospitalenvironment, and can also be difficult to adapt to the rehabilitationrequirements of a particular patient since the range of permittedmovements is often confined by a mechanical system.

US 2011/0054870 discloses a VR based system for rehabilitation of apatient, wherein a position of a body part of a patient is tracked by amotion camera. Software is used to create a motion avatar, which isdisplayed to the patient on a monitor. In an example, if a patient movesonly a right arm when movement of both arms are prescribed, then theavatar can also display motion of the left arm.

A similar system is disclosed in ‘The design of a real-time, multimodalbiofeedback system for stroke patient rehabilitation’, Chen, Y et al.,ACM International Conference on Multimedia, 23 Oct. 2006, whereininfra-red cameras are used to track a 3-dimensional position of markerson an arm of a patient. Using a monitor, in VR a position of the arm ofthe patient is displayed as predefined movement patterns are completed,such as the grasping of a displayed image.

A drawback of certain VR based systems is that they only measure theresponse of the body part to an instructed task. Accordingly, they donot directly measure cortical activity in response to a displayedmovement of a body part, only the way in which an area of the brain cancontrol a body part. This may lead to areas of the brain being treatedother than those which are damaged, or at least an inability to directlymonitors a particular area of the brain. Moreover, the patient is notfully immersed in the VR environment since they look to a separatemonitor screen to view the VR environment.

In WO 2011/123059 and US 2013/046206, VR based systems with brainmonitoring and motion tracking are described, the main drawback of knownsystems being that they do not reliably nor accurately controlsynchronization between stimulation or action signals and brain activitysignals, which may lead to incorrect or inaccurate processing and readout of brain response signals as a function of stimuli or actions.

In conventional systems, in order to synchronize multimodal data(including physiological, behavioral, environmental, multimedia andhaptic, among others) with stimulation sources (e.g., display, audio,electrical or magnetic stimulation) several independent, dedicated (i.e.for each data source) units are connected in a decentralized fashion,meaning that each unit brings its inherent properties (module latenciesand jitters) into the system. Additionally, these units may havedifferent clocks, therefore acquiring heterogeneous data with differentformats and at different speeds. In particular, there is nocomprehensive system that comprises stereoscopic display of virtualand/or augmented reality information, where some content may be relatedto some extent to the physiological/behavioral activity of any relateduser and registered by the system, and/or any information coming fromthe environment. Not fulfilling the abovementioned requirements may havenegative consequences in various cases in different application fields,as briefly mentioned in the following non-exhaustive list of examples:

-   -   a) Analysis of neural responses to stimulus presentation is of        importance in many applied neuro-science fields. Current        solutions compromise the synchronization quality, especially in        the amount of jitter between the measured neural signal (e.g.,        EEG) and the simulation signal (e.g., display of a cue). Due to        this, not only the signal to noise ratio of acquired signals is        lowered but also limit the analysis to lower frequencies        (typically less than 30 Hz). A better synchronization ensuring        least jitter would open up new possibilities of neural signals        exploration in the higher frequencies as well as precise (sub        millisecond) timing based stimulation (not only non-invasive        stimulation, but also invasive stimulation directly at the        neural cite and subcutaneous stimulation).    -   b) Virtual reality and body perception: If the synchronization        between the capture of user's movements and their mapping onto a        virtual character (avatar) that reproduces the movement in real        time is not achieved, then, the delayed visual feedback of the        performed movement via a screen or head-mounted display will        give to the user the feeling that he/she is not the author of        such movement. This may have important consequences in motor        rehabilitation, where patients are trained to recover mobility,        as well as for training or execution of extremely dangerous        operation as deactivating a bomb by manipulating a robot        remotely.    -   c) Brain-computer interfaces: If the synchronization between        motor intention (as registered by electroencephalographic data),        muscle activity and the output towards a brain body-controlled        neuroprosthesis fails, it is not possible to link motor actions        with neural activation, preventing knowledge about the neural        mechanisms underlying motor actions necessary to successfully        control the neuroprosthesis.    -   d) Neurological examinations: The spectrum of        electroencephalographic (EEG) data may reach up to 100 Hz for        superficial, non-invasive recordings. In such a case, the time        resolution is in the range of tens of milliseconds. If the        synchronization between EEG and events evoking specific brain        responses (e.g. P300 response for a determined action happening        in virtual environments) fails, then it is not possible to        relate the brain response to the particular event that elicited        it.    -   (e) Functional re-innervation training to use a sophisticated        neuroprosthesis device by an amputee patient: A hybrid        brain-computer interface (BCI) system coupled with FES and        sub-cutaneous stimulation may be used in elaborating and        optimizing functional re-innervation into residual muscles        around stumps or other body parts of an amputees. For optimal        results, it is important to have high quality synchronization        between the sensor data and stimulation data for generating        precise stimulation parameters.

In non-medical applications, especially in consumer applications such asgaming or sports activity training, high portability and ease of use ofsensing and feedback systems are important factors.

The cost of devices is of course also an important issue in consumerapplications. In medical applications the cost of diagnosis or treatmentis important, whereby such costs are affected not only by the cost ofdevices used per se, but also the cost of using the devices, including:ease of set up, manipulation and of obtaining and interpreting results;reuse and sterilization.

The placement of EEG electrodes have been widely researched and one ofthe commonly used models is the so called “10-20 electrode placementsystem”. Conventional head-mounted electroencephalogram (EEG) sensingdevices used in medical or research applications typically comprise atextile cap with a plurality of sensors clipped to the cap. The captypically is provided with orifices or other fixing means in thepositions corresponding to the electrode placement model used. Theelectrodes are then wired to an amplifying and computing device. Chinstraps are provided to stably and accurately adjust and position the capso that the electrodes are correctly positioned and pressed against theuser's head. The positioning of the cap, connection of electrodes andthe chip strap or other fixing devices render such conventional systemsuninteresting for many consumer applications and tedious even formedical applications. Drawbacks include: not visually aesthetic;overheating of the electrode contact due to reduced air-flow;time-consuming placement of the cap because of adjustments to anatomicalinion-nasion distance to ensure 10-20 system and complex set-up (e.g.,placement of electrodes, contacts, connectivity etc.); the need forseparate electronics and cap electrodes to allow washing of theelectrodes.

A medical EEG head cap that avoids individual wire connections bymounting and connecting electrodes to a flexible circuit is disclosed inU.S. Pat. No. 4,967,038 and U.S. Pat. No. 5,038,782. Another medical EEGheadpiece with elastic straps is disclosed in EP0541393. All three ofthese prior systems suffer from many of the drawbacks mentioned above.

Further EEG head-mounted devices are disclosed in US 2015/0011857, US2013/0172721, WO 2013/124366, US2011/0282231, and US2013/0303874,however that also suffer at least some of the drawbacks including thedifficulty to adjust for different head sizes, non-optimal placement ofelectrodes, difficulty to allow washing of the electrodes/head set forre-use, discomfort to wear and/or high manufacturing costs.

SUMMARY OF THE INVENTION

It is an aim of the invention to provide a brain activity measurementand feedback system that is convenient and comfortable to wear,reliable, and simple to use.

It would be advantages to provide a system that is easily washable andsterilizable.

It would be advantageous to provide a system that saves time forplacement and operation.

It would be advantageous to provide a brain activity measurement andfeedback system that may be adapted for home use, for ambulatoryapplications, or for mobile applications.

It would be advantageous to provide a system which is cost effective tomanufacture and to use.

It would be advantageous to provide a brain activity measurement andfeedback system that provides a user with a virtual or augmented realityenvironment that can be utilized to improve the response of thecognitive and sensory motor system, for instance in the treatment ofbrain damage or in the training of motor skills.

It would be advantageous to provide a physiological parametermeasurement and motion tracking system (e.g., movements head and body)that ensures accurate real time integration of measurement and controlof physiological stimuli and response signals.

It would be advantageous to provide a brain activity measurement andfeedback system that can generate a plurality of stimuli signals ofdifferent sources (e.g. visual, auditive, touch sensory, electric,magnetic) and/or that can additionally measure a plurality ofphysiological response signals of different types (e.g. body partmovement, eye movement, galvanic skin response).

It would be advantageous to reduce electrical interference among theinput modules (measurements) and output modules (stimuli) and systemoperation.

It would be advantageous to easily adapt the system to various head andbody sizes.

It would be advantageous to provide a more immersive VR experience.

It would be advantageous to ensure high quality electrical orelectrochemical contact between skin and electrode tip.

It would be advantageous not to have to remove an electrode sensing netwhen a user needs to take-off a head-mounted display assembly in orderto maintain electrical or electrochemical contacts are maintained.

Disclosed herein is a head set comprising an EEG sensing devicecomprising EEG sensors configured to be mounted on a head of a wearer soas to position the EEG sensors at selected positions of interest overthe wearers scalp. The EEG sensing device comprises a sensor support anda flexible electronic circuit assembled to the sensor support, the EEGsensors connected to the flexible electronic circuit. The sensor supportand flexible circuit comprise a central stem configured to extend alonga center plane of the top of the head in a direction from nasion toinion, a front lateral branch extending laterally from the central stemconfigured to extend across a front portion of a wearer's head, a centerlateral branch extending laterally from the central stem configured toextend across a top portion of a wearer's head essentially between thewearer's ears, and a rear lateral branch extending laterally from thecentral stem configured to extend across a back portion of a wearer'shead.

According to a first aspect of the invention, the sensor supportcomprises a base wall and side walls extending along edges of the basewall to form an essentially flat “U” shaped channel in which theflexible circuit is assembled.

According to a second aspect of the invention, especially forembodiments using wet electrodes, the flexible circuit comprisesorifices adjacent the EEG sensor contacts or electrodes, wherein the EEGsensor orifices overlap the flexible circuit orifices such that athrough passage between a top surface and a bottom surface of the sensorsupport is provided to allow conductive gel to be inserted from a topside of the head set while it is positioned on a wearer's head.

The through passages could also serve, alternatively or in addition, asaccess passages for inserting complementary electrodes or other sensors,for instance to test or calibrate the electrodes of the head set, or toadd additional sensors or stimulation devices, such as temperaturesensors and NIRS, tDCS, tRNS, tACS devices. In a particular embodiment,the through passages may advantageously be used for access by NearInfrared Light emitters and receivers (NIRS) placed over or in thesethrough passages or in other extra orifices. Near infrared spectroscopycan be very useful to measure blood oxygen levels of the brain tissue.This way one can co-register the electrical and blood-oxygen levels.

Each of the lateral branches further comprises extensions extending in afront to rear, or in a rear to front direction. The EEG sensors arepositioned in discrete spaced apart positions along the stem, branchesand extensions, for instance in positions according to the international10-20 system, or according to other EEG positioning systems.

According to a third aspect of the invention, the center lateralbranches comprise extensions extending both in a front to rear and in arear to front direction such that the center lateral branches can betensioned to both the front lateral branches and the rear lateralbranches.

In a preferred embodiment a flexible sealing material is filled over theflexible circuit in the channel in order to seal the electrical circuittracks and components on the flexible circuit within the channel.

In a preferred embodiment the sensor support is a single piece part.

In a preferred embodiment the sensor support is molded or formed from aflexible polymeric material.

In a preferred embodiment the flexible circuit comprises a single pieceflexible substrate.

The head set sensor support may further comprise tensioner anchorsconfigured to anchor elastic tensioners between positions in the stem,branches and extensions of the EEG sensing device, and also between theEEG sensing device and a head mount frame support.

A base wall of the sensor support comprises EEG sensor orifices to allowaccess to the EEG sensor contacts or electrodes on the flexible circuit.

In an advantageous embodiment, each EEG sensor on the flexible circuitis positioned a discrete EEG signal amplifier configured to amplify thebrain electrical activity signal picked up by the corresponding EEGsensor. The EEG sensor may comprise according to the variant, comprisepassive or active sensors.

The EEG sensors may comprise electrodes in the form of conductivecircuit pads on a surface of a substrate of the flexible circuitintended to face the wearers scalp, or protruding conductivecompressible elements mounted on the flexible substrate and electricallyconnected to a circuit trace of a substrate of the flexible circuit. TheEEG sensors may contact the scalp through a wet contact, e.g. using asoft gel, or a semi-solid structure acting in a similar manner to a wetcontact, or a dry contact depending on the variant. Furthermore, EEGsensors according to certain variants may not be in direct contact withthe scalp, for instance sensors based on near infrared light orcapacitive charging

In certain embodiments the head set may comprise a head-mounted display(HMD) fixed to a head mount frame support and configured to bepositioned over the eyes of a wearer of the headset.

The HMD may comprise a display unit having display means in the form ofan electronic screen configured for positioning in front of the wearer'seyes to present visual information to the wearer and optionally furthercomponents providing feedback, stimulation or information to the wearer.

The HMD may house various sensing devices and information capture andtransmission devices, such as one or more cameras, depth sensors, headmovement sensing unit, wireless communication device to interconnect theheadset to external electronic devices and computing systems in awireless fashion, and an on-board power supply for autonomous operationof the head set.

In certain embodiments, the headset may further incorporate a pluralityof sensors configured to measure different physiological parameters,selected from a group consisting of Electrocorticogram (ECOG) sensors,eye movement sensors, and head movement sensing unit.

In certain embodiments, the headset may advantageously furtherincorporate one or a plurality of brain or nerve stimulation devices,for instance Functional Electrical Stimulation (FES) devices, comprisingfor instance electrodes configured for trans-cranial alternating currentstimulation (tACS), direct current stimulation (tDCS), trans-cranialmagnetic stimulation (TMS) and trans-cranial ultrasonic stimulation.

In certain embodiments, the headset may further incorporate aposition/motion detection system operable to detect a position/motion ofa body part of a user, the position/motion/detection system comprisingone or more colour cameras, and a depth sensor.

Also disclosed herein is a physiological parameter measurement systemcomprising a control system, a sensing system, and a stimulation system,the sensing system comprising one or more physiological sensorsincluding at least brain electrical activity sensors mounted in the headset. The stimulation system may comprise one or more stimulation devicesincluding at least a visual stimulation system. The control system maycomprising an acquisition module configured to receive sensor signalsfrom the sensing system, and a control module configured to process thesignals from the acquisition module and control the generation ofstimulation signals to one or more devices of the stimulation system.The control system further comprises a clock module wherein the controlsystem is configured to time stamp signals related to the stimulationsignals and the sensor signals with a clock signal from the clockmodule, enabling the stimulation signals to be synchronized with thesensor signals by means of the time stamps.

The time stamped signals related to the stimulation signals may becontent code signals received from the stimulation system.

The system may advantageously comprise a display register configured toreceive display content representing a final stage before the displaycontent is activated on the display, the display register beingconfigured to generate a display content code signal for transmission tothe control system, a time stamp being attached to the display contentcode signal by the clock module.

The sensing system may comprise physiological sensors selected from agroup comprising Electromyogram (EMG) sensors, Electrooculography (EOG)sensors, Electrocardiogram (ECG) sensors, Inertial Sensors (INS), Bodytemperature sensor, Galvanic skin sensor, pulse oximetry sensor, andrespiration sensors.

The stimulation system may comprise stimulation devices selected from agroup comprising audio stimulation device, Functional ElectricalStimulation (FES) devices, and haptic feedback devices, said functionalelectrical stimulation devices being connected to the control system andoperable to electrically stimulate one or more body parts of the user.The FES devices may be selected from a group consisting of electrodesconfigured to stimulate nerves or muscles, trans-cranial alternatingcurrent stimulation (tACS), direct current stimulation (tDCS),trans-cranial magnetic stimulation (TMS) and trans-cranial ultrasonicstimulation.

Each stimulation device may comprise an embedded sensor whose signal isregistered by a synchronization device.

The system may further comprise in an embodiment, a robotic system fordriving movements of a limb of the user and configured to provide hapticfeedback.

The clock module may be configured to be synchronized with clock moduleof other systems, including external computers.

The system may advantageously comprise an exercise logic unit configuredto generate visual display frames including instructions and challengesto the display unit.

The system may advantageously comprise an events manager unit configuredto generate and transmit stimulation parameters to the stimulation unit.

Further objects and advantageous features of the invention will beapparent from the claims, from the detailed description, and annexeddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the invention, and to show how embodimentsof the same may be carried into effect, reference will now be made, byway of example, to the accompanying diagrammatic drawings in which:

FIGS. 1a and 1b are schematic illustrations of prior art systems;

FIG. 2a is a schematic diagram illustrating an embodiment of theinvention in which display content displayed to a user is synchronizedwith response signals (e.g. brain activity signals) measured from theuser;

FIG. 2b is a schematic diagram illustrating an embodiment of theinvention in which audio content played to a user is synchronized withresponse signals (e.g. brain activity signals) measured from the user;

FIG. 2c is a schematic diagram illustrating an embodiment of theinvention in which a plurality of signals applied to a user aresynchronized with response signals (e.g. brain activity signals)measured from the user;

FIG. 2d is a schematic diagram illustrating an embodiment of theinvention in which a haptic feedback system is included;

FIG. 2e is a schematic diagram illustrating an embodiment of theinvention in which a neuro-stimulation signal is applied to a user;

FIG. 3a is a simplified schematic diagram of an embodiment of aphysiological parameter measurement system according to the invention;

FIG. 3b is a detailed schematic diagram of a control system of thesystem of FIG. 3 a;

FIG. 3c is a detailed schematic diagram of a physiological trackingmodule of the control system of FIG. 3 b;

FIG. 4a is a perspective view of an embodiment of a head-mounted displayunit of a headset according to an embodiment of the invention;

FIGS. 4b and 4c are front and back perspective views of a headsetaccording to an embodiment of the invention, mounted on a head of auser;

FIG. 4d is a schematic top view of a headset according to an embodimentof the invention, mounted on a head of a user;

FIGS. 4e and 4f are bottom and side views of an EEG sensing device of ahead set according to an embodiment of the invention;

FIGS. 4g and 4h are perspective views of a portion of a support of theEEG sensing device according to an embodiment of the invention;

FIG. 4i is a top side view of a flexible circuit of the EEG sensingdevice according to an embodiment of the invention;

FIG. 4j is a bottom side view of a flexible circuit of the EEG sensingdevice according to an embodiment of the invention;

FIG. 5 is a top view illustrating an arrangement ofelectroencephalography locations and their nomenclature according to theinternational 10-20 system;

FIG. 6 is a front view of an exemplary arrangement of EMG sensors on abody of a user;

FIG. 7 is a diagrammatic view of a process for training a stroke victimusing an embodiment of the system;

FIG. 8 is a view of screen shots which are displayed to a user duringthe process of FIG. 7;

FIG. 9 is a perspective view of a physical setup of a physiologicalparameter measurement system according to an exemplary embodiment of theinvention;

FIG. 10 is a schematic block diagram of an example stimulus and feedbacktrial of a physiological parameter measurement system according to anexemplary embodiment of the invention;

FIG. 11 is a schematic block diagram of an acquisition module of aphysiological parameter measurement system according to an exemplaryembodiment of the invention;

FIG. 12 is a diagram illustrating time stamping of a signal by a clockmodule of a physiological parameter measurement system according to anexemplary embodiment of the invention;

FIG. 13 is a data-flow diagram illustrating a method of processingphysiological signal data in a control system of a physiologicalparameter measurement system according to an exemplary embodiment of theinvention;

FIG. 14 is a flowchart diagram illustrating a method of processingevents in a control system of a physiological parameter measurementsystem according to an exemplary embodiment of the invention;

FIG. 15a is a bottom view of a portion of the headset according to anembodiment of the invention, showing an electrode orifice in detail;

FIG. 15b is a top view of the portion of the headset of FIG. 15 a;

FIG. 15c is a cross-sectional view through line A-A of FIG. 15 a;

FIG. 15d is a perspective cross-sectional view through line D-D of FIG.15 b;

FIG. 15e is a cross-sectional view of the portion of headset of FIG. 15amounted on a head of a user.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Referring to the figures, a physiological parameter measurement andmotion tracking system according to embodiments of the inventiongenerally comprises a control system 12, a sensing system 13, and astimulation system 17.

The sensing system comprises one or more physiological sensors includingat least brain electrical activity sensors, for instance in the form ofelectroencephalogram (EEG) sensors 22. The sensing system may comprisesother physiological sensors selected from a group comprisingElectromyogram (EMG) sensors 24 connected to muscles in user's body,Electrooculography (EOG) sensors 25 (eye movement sensors),Electrocardiogram (ECG) sensors 27, Inertial Sensors (INS) 29 mounted onthe user's head and optionally on other body parts such as the userslimbs, Body temperature sensor, Galvanic skin sensor. The sensing systemfurther comprises position and/or motion sensors to determine theposition and/or the movement of a body part of the user. Position andmotion sensors may further be configured to measure the position and/ormovement of an object in the field of vision of the user. It may benoted that the notion of position and motion is related to the extentthat motion can be determined from a change in position. In embodimentsof the invention, position sensors may be used to determine bothposition and motion of an object or body part, or a motion sensor (suchas an inertial sensor) may be used to measure movement of a body part orobject without necessarily computing the position thereof. In anadvantageous embodiment at least one position/motion sensor comprises acamera 30 and optionally a distance sensor 28, mounted on a head set 2configured to be worn by the user.

The Stimulation system 17 comprises one or more stimulation devicesincluding at least a visual stimulation system 32. The stimulationsystem may comprise other stimulation devices selected from a groupcomprising audio stimulation device 33, and Functional ElectricalStimulation (FES) devices 31 connected to the user (for instance tostimulate nerves, or muscles, or parts of the user's brain e.g. tostimulate movement of a limb), and haptic feedback devices (for instancea robot arm that a user can grasp with his hand and that provides theuser with haptic feedback). The stimulation system may further compriseAnalogue to Digital Converters (ADC) 37 a and Digital to AnalogueConverters (DAC) 37 b for transfer and processing of signals by acontrol module 51 of the control system. Devices of the stimulationsystem may further advantageously comprise means to generate contentcode signals 39 fed back to the control system 12 in order to timestampsaid content code signals and to synchronize the stimulation signalswith the measurement signals generated by the sensors of the sensingsystem.

The control system 12 comprises a clock module 106 and an acquisitionmodule 53 configured to receive content code signals from thestimulation system and sensor signals from the sensing system and totime stamp these signals with a clock signal from the clock module. Thecontrol system further comprises a control module that processes thesignals from the acquisition module and controls the output of thestimulation signals to devices of the stimulation system. The controlmodule further comprises a memory 55 to store measurement results,control parameters and other information useful for operation of thephysiological parameter measurement and motion tracking system.

FIG. 3a is a simplified schematic diagram of a physiological parametermeasurement and motion tracking system 10 according to an embodiment ofthe invention. The system 10 comprises a control system 12 which may beconnected to one or more of the following units: a physiologicalparameter sensing system 14; position/motion detection system 16; and ahead set 2, all of which will be described in more detail in thefollowing.

The physiological parameter sensing system 14 comprises one or moresensors 20 configured to measure a physiological parameter of a user.The sensors 20 comprise a plurality of electroencephalogram (EEG)sensors 22 mounted in the head set configured to measure corticalactivity of a user by measuring electrical activity in a brain of auser. EEG sensors measure voltage fluctuations result from ionic currentflows within the neurons of the brain.

FIG. 5 shows a top view of schematized head showing a known nomenclatureof locations for electrode arrangement corresponding to theinternational 10-20 system, wherein the shaded groups comprise zones ofprimary interest in many uses of embodiments of the invention.

FIGS. 4a, 4b and 4c shows an arrangement of electroencephalogram sensors22 of a headset according to an embodiment of the invention, positionedon a head of a user.

The head set 2 according to an embodiment of the invention comprises ahead mount frame support 9 configure to position and hold the head seton a user's head, and a brain activity (EEG) sensing device 3 attachedto the head mount frame support.

The head set may, in certain embodiments, further comprise ahead-mounted display 19 fixed to the head mount frame support, Thehead-mounted display (HMD) 19 is fixed to the head mount frame supportand configured to be positioned over the eyes of a wearer of theheadset. The HMD 19 may in embodiments be removably attachable to thehead mount frame support.

The HMD 19 comprises a display unit 32 mounted in a display unit support36, the display unit having display means in the form of an electronicscreen configured for positioning in front of the wearer's eyes topresent visual information to the wearer. The HMD may further compriseother components providing feedback, stimulation or information to thewearer, such as an audio unit 33 to generate sound for the wearer. TheHMD may further conveniently house electronics and support furtherinformation capture devices such as one or more cameras 30, depthsensors 28, microphone, and various sensors such as a head movementsensing unit 40, 50, an eye gaze sensing unit 100. The HMD may furtherconveniently house wireless communication devices for wirelesscommunication using any one or more on various known wirelesscommunication protocols to interconnect the headset to externalelectronic devices and computing systems in a wireless fashion. The HMDmay advantageously further house an on-board power supply, in the formof a battery, for autonomous operation of the head set. The head set mayalso advantageously be provided with one, or a plurality of connectorsconfigured for connecting the head set to external power supply andcomputing systems or sensors. Further description of various sensors andcomponents of the HMD will be provided further on in this specification.

The EEG sensing device 3 is fixed to the head mount frame support andconfigured to be positioned over the head of a wearer of the headset soas to position electrodes over the wearers scalp at the selectedpositions of interest for the application concerned (e.g. for medicalapplications, for gaming applications, for training applications, forresearch applications, for brain monitoring applications). In apreferred embodiment the electrode positions correspond to positionsknown from the international 10-20 system, however other systems may beused and electrode positions refined as a function of the results ofresearch on brain electrical activity.

The EEG sensing device 3 comprises an EEG sensor support 4 and aflexible circuit 6 assembled to the EEG sensor support 4.

The flexible circuit 6 is configured to capture and process brainactivity electrical signals and comprises a flexible circuit substrate601 on or in which are mounted EEG sensors 22 and an EEG signalprocessing circuit 8 comprising electronic components 8 a and circuittraces 8 b interconnecting components and sensors and interconnectingthe flexible circuit to external power supply and circuits via aconnection portion 41 at an end of the flexible circuit. In anadvantageous embodiment, the flexible circuit comprises a pluggableelectrical connector for plugging to a complementary pluggableelectrical connector on the HMD. A pluggable connector 41 isparticularly advantageous in that it allows the user to be able take offthe HMD without having to remove the electrode head set. This allows theelectrical or electrochemical contacts of the electrodes to bemaintained in position and undisturbed on the wearer's scalp.

The flexible circuit substrate may advantageously be provided in theform of a thin flexible semi-rigid polymer substrate per se well knownin the art of flexible printed circuit technology. The plurality of EEGsensors 22 comprise electrodes spatially positioned on the flexiblecircuit at the desired electrical brain activity measurement positions,and next to each electrode is positioned a discrete EEG signal amplifier8 b. The close proximity of the amplifier to the electrode allows thebrain electrical activity picked up by the electrode to be amplifiedlocally before transmission to a signal analyzing circuit in order toimprove the signal to noise ratio of the measured brain activitysignals.

The flexible circuit 6 and support 4 of the EEG sensing device areconfigured such that when they are bent to conforms to a generallyspherical or ellipsoid three dimensional form corresponding essentiallyto the general morphology of a top half of a human head, the EEG sensorsare positioned accurately in the desired positions according to thechosen sensor placement. Accuracy and ease of positioning the electrodescorrectly and quickly is provided inter alia by the advantageous shapeof the sensing device 3, and in particular of the shape of the support 4and the flexible circuit 6 mounted in the support.

In an advantageous embodiment the support comprises a central stem 4 aconfigured to extend along a center plane of the top of the head in adirection from a nose to a centre of the back of a wearers head, andextending laterally from the central stem 4 a, a front lateral branch 4b configured to extend across a front portion of a wearer's head, acenter lateral branch 4 c configured to extend across a top portion of awearer's head essentially between the wearer's ears, and a rear lateralbranch 4 d configured to extend across a back portion of a wearer'shead. Each of the lateral branches further comprise extensions,including back and side extensions 4 b 1, 4 b 2, front and rearextensions 4 c 1, 4 c 2 and front and side extensions 4 d 1, 4 d 2respectively. The flexible substrate 6 has a shape that conforms to thesupport and thus comprises a central stem 6 a, a front lateral branch 6b, a center lateral branch 6 c, and a rear lateral branch 6 d that aremounted in corresponding portions of the support. Each of the lateralbranches further comprises extensions, including back and sideextensions 6 b 1, 6 b 2, front and rear extensions 6 c 1, 6 c 2 andfront and side extensions 6 d 1, 6 d 2 respectively that are mounted incorresponding extensions of the support 4. EEG sensors are positioned indiscrete spaced apart positions along the stem, branches and extensions.As best illustrated in FIG. 4d , the stem, branches and extensions areconfigured in this example to position EEG sensors at certain positionsof interest according to the international 10-20 system.

The support 4 may advantageously be made of a flexible elastic orsemi-rigid material (as opposed to a textile fabric used in conventionalcaps) that is flexible enough to bend out of its major plane to conformto the rounded shape of a wearer's head, but that has sufficientself-supporting rigidity in conjunction with the flexible circuitsubstrate mounted therein, in a direction orthogonal to the bendingplane, to generally keep its shape between branches. In effect, the thinflexible circuit substrate is very flexible in the direction Bperpendicular to the major surface of the substrate but relatively rigidagainst bending in a direction P parallel to the major surface. Inadvantageous embodiments the support may be molded in a single piece,for instance injection molded, press die molded, or blow-molded, out ofan elastic polymeric material such as a silicon rubber polymer orsimilar. The support comprises a base wall 401 and side walls 402extending along edges of the base wall to form an essentially flat “U”shaped channel 403 in which the flexible circuit 6 is inserted. Aflexible sealing material 5, for instance a polymeric elastic pottingresin or silicon rubber based potting material may be filled over theflexible circuit in the channel, thus forming a top wall of the support,in order to seal the electrical circuit tracks and components on theflexible circuit. This advantageously allows the EEG sensing device 3 ofthe head set 2 to be resistant to liquids and to be easily washable andif needed sterilizable, without damaging the electronic circuitscontained therein. A flexible top wall of a similar or differentmaterial to the bottom wall could also be assembled over the flexiblecircuit and bonded, welded or fastened with mechanical means to the basewall.

As best seen in FIGS. 4g, 4h and 15a-15e , the base wall 401 of thesupport advantageously comprises EEG sensor orifices 404 to allow accessto the EEG sensor contacts or electrodes 221 on the flexible circuitsubstrate. In an embodiment, the EEG sensor orifices may comprise afirst portion 404 a below the electrode 221 and a second portion 404 bthat is aligned with or overlaps an orifice 602 in the flexible circuit,such that a through passage is formed extending from the bottom surface406 to the top surface 506 of the support 4. The through passageprovides access from the top of the sensing device to the scalp of thewearer to inject a conductive gel 37 from the top of the device when thesensing device is mounted on a wearer's scalp. The conductive gel 37provides a good electrical contact between the wearer's skin and theelectrodes 221. Alternatively, a gel- or water-based electrode could bere-wetted by injecting water in the through passages without removingthe sensing device. As best illustrated in FIG. 15e , the gel 37 is ableto be injected in the second portion 404 a of the orifice and spreadlaterally to fill the second portion 404 b of the orifice 404 to contactthe electrode 221 arranged on a bottom side of the flexible circuit.

The through passages could also serve, alternatively or in addition, asaccess passages for inserting complementary electrodes or other sensors,for instance to test or calibrate the electrodes of the head set, or toadd additional sensors or stimulation devices, such as temperaturesensors and NIRS, tDCS, tRNS, tACS devices. In a particular embodiment,the through passages may advantageously be used for access by NearInfrared Light emitters and receivers (NIRS) placed over or in thesethrough passages or in other extra orifices. Near infrared spectroscopycan be very useful to measure blood oxygen levels of the brain tissue.This way one can co-register the electrical and blood-oxygen levels.

The support 4 further comprises tensioner anchors 405, for instance inthe form of fixing orifices, that allow elastic ties 7 or other forms ofelastic tensioners to be anchored between positions in the stem,branches and extensions of the EEG sensing device, and also between theEEG sensing device and the head mount frame support 9. The flexiblecircuit substrate 601 may be provided with corresponding tensionerorifices 605 aligned with the orifices 405 in the support. The elastictensioners 7 apply a certain tension in the structure to ensure that thebranches and extensions of the sensing device 3 conform in a snug fit tothe shape of a wearer's head and further to ensure the correct positionof the various EEG sensors 22 on the wearers scalp. In particular, thestem and branch arrangement of the sensing device ensures the correctposition of the front, rear and central electrodes, whereby the elastictensioners allow a certain adjustment in the distance between center andfront, and between center and rear, to conform to different head sizes.The sensing device may be provided in various sizes, whereby with threedifferent sizes more than 95% of the range of adult human head sizes maybe covered with accurate electrode placement. For children additionalsizes may be provided. The different sizes may concern essentially thelength of the central stem and central lateral branch. In order to covermost anatomical ranges, Anchors 405 are positioned at extremities ofeach branch and extension, and also in opposing positions withinbranches to anchor the opposing end of an elastic tensioner 7 connectedto an extremity, as best illustrated for instance in FIGS. 4b, 4c, and4d . The length of each elastic ties may be pre-adjusted during assemblyof the sensing device and thereafter remain fixed lengths, but in someembodiments some or all of the elastic tensioners may be of adjustablelengths. The elastic tensioners may be provided in different forms, forinstance in the form of an elastic textile based band.

In an alternative embodiment, some or all of the tensioners 7 could beintegrally formed with the support 4 as a single part. For instance someor all of the tensioners could be molded with the support 4, forinstance in the same flexible polymeric material as the support. In suchembodiment, the elastic tensioners may be provided with a thinner wallor smaller width, or with serpentine sections so as to increase theelasticity of the elastic tensioners in comparison to the supportportion. Also, during the forming process of the integral support 4 andtensioners 7 part, different materials may be incorporated in differentportions, for instance by overmolding or by welding inserts of anothermaterial in the injected or molded main support material, or forinstance by dual component injection molding.

The head mount frame support 9 may have various configurations, itsfunction being to support the sensing device under a certain elastictension (as described above) and as a function of the embodiment, tofurther support the head-mounted display. In an embodiment, asschematically illustrated, the head mount frame support 9 may comprise ahead band with an adjustable diameter to allow a correct fit around awearer's head, the head band being rigid against bending at least in thepulling direction of the elastic tensioners 7. Other head mountstructures including helmet like structures to support the HMD andsensing device may be used within the scope of this invention.

In an advantageous embodiment, the EEG electrode may be in the form of aconductive circuit pad on a surface of the flexible circuit substrateintended to face the wearer's scalp. A liquid or self-supportingconductive gel or solid gel may be position on the electrode for contactwith the wearer's scalp. In another embodiment, the conductive bridgebetween the electrical contact on the flexible circuit substrate andscalp of the wearer may be provided by a conductive bead, stud or otherprotruding element for a dry electrode or wet electrode contact theagainst the wearer's scalp. The EEG electrode may also be in the form ofa protruding conductive compressible element, for instance a stamped andformed sheet metal contact, mounted on the flexible substrate andelectrically connected to a circuit trace of the substrate by crimping,welding, soldering or other per se known techniques for connectingelectronic components on a flexible circuit board.

EEG sensors according to variants of the invention may include sensorsthat do not require any direct electrical contact with the scalp (orskin) and may in particular include sensors based on (i) near infraredlight and (ii) capacitive charging, according to the per se knownmeasurement principles discussed below:

(i) Near Infrared Spectroscopy (NIRS) uses the fact that thetransmission and absorption of NIR light in human body tissues containsinformation about haemoglobin concentration changes. When a specificarea of the brain is activated, the localized blood volume in that areachanges quickly. Optical imaging can measure the location and activityof specific regions of the brain by continuously monitoring bloodhaemoglobin levels through the determination of optical absorptioncoefficients. NIRS can be used for non-invasive assessment of brainfunction through the intact skull in human subjects by detecting changesin blood haemoglobin concentrations associated with neural activity.

(ii) Capacitive electrodes are based on the measurement of chargedisplacements on the body surface due to brain activity. This change ofthe charge can in turn affect the charge on a metal plate close to thebody. Since this electrical plate does not require direct electricalcontact to the body, it can be isolated from the body. The measurementof the capacitive EEG (cEEG) is therefore also possible through thehair. A supersensitive signal amplifier is preferably connected to thisplate which amplifies the brain signal and processes it.

The one or more sensors 20 may additionally comprise sensors 24configured to measure movement of a muscle of a user, for example bymeasuring electrical potential generated by muscle cells when the cellsare electrically or neurologically activated. A suitable sensor is anelectromyogram EMG sensor. The sensors 24 may be mounted on variousparts of a body of a user to capture a particular muscular action. Forexample for a reaching task, they may be arranged on one or more of thehand, arm and chest. FIG. 6 shows an exemplary sensor arrangement,wherein the sensors 24 are arranged on the body in: a first group 24 aon the biceps muscle; a second group 24 b on the triceps muscle; and athird group 24 c on the pectoral muscle.

The one or more sensors 20 may additionally comprise sensors 25configured to measure electrical potential due to eye movement. Asuitable sensor is an electrooculography (EOG) sensor. In an embodiment,as shown in FIG. 4a , there are four sensors that may be arranged inoperational proximity to the eye of the user. However it will beappreciated that other numbers of sensors may be used. In anadvantageous embodiment the sensors 25 are conveniently connected to adisplay unit support 36 of the head set, for example they are affixedthereto or embedded therein.

The sensors 20 may alternatively or additionally comprise one or more ofthe following sensors: electrocorticogram (ECOG); electrocardiogram(ECG); galvanic skin response (GSR) sensor; respiration sensor;pulse-oximetry sensor; temperature sensor; single unit and multi-unitrecording chips for measuring neuron response using a microelectrodesystem. It will be appreciated that sensors 20 may be invasive (forexample ECOG, single unit and multi-unit recording chips) ornon-invasive (for example EEG). Pulse-oximetry sensor is used formonitoring a patient's oxygen saturation, usually placed on finger tipand may be used to monitor the status of the patient. This signal isparticularly useful with patients under intensive care or special careafter recovery from cardio-vascular issues. It will be appreciated thatfor an embodiment with ECG and/or respiration sensors, the informationprovided by the sensors may be processes to enable tracking of progressof a user. The information may also be processed in combination with EEGinformation to predict events corresponding to a state of the user, suchas the movement of a body part of the user prior to movement occurring.It will be appreciated that for an embodiment with GSR sensors, theinformation provided by the sensors may be processed to give anindication of an emotional state of a user. For example, the informationmay be used during the appended example to measure the level ofmotivation of a user during the task.

In an advantageous embodiment, the head set 2 of the physiologicalparameter sensing system 14 comprises a wireless transceiver which isoperable to wirelessly transfer data to external devices, for instanceto a wireless transceiver of the physiological parameter processingmodule 54, or to a wireless transceiver of the skeletal tracking module52.

The position/motion detection system 16 comprises one or more sensors 26suitable for tracking motion of the skeletal structure or a user, orpart of the skeletal structure such as an arm. In an advantageousembodiment the sensors comprise one or more cameras which may bearranged separate from the user or attached to the head set 2. At leastone, or each camera is arranged to capture the movement of a user andpass the image stream to a skeletal tracking module which will bedescribed in more detail in the following.

In an embodiment the sensors 26 may comprise three cameras: two colourcameras 28 a, 28 b and a depth sensor camera 30. However, in anotherembodiment there may be one colour camera 28 and a depth sensor 30. Asuitable colour camera may for instance have a resolution of VGA 640×480pixels and a frame rate of at least 60 frames per second. The field ofview of the camera may also be matched to that of the head-mounteddisplay, as will be discussed in more detail in the following. Asuitable depth camera may have a resolution of QQ VGA 160×120 pixels.

In an advantageous embodiment two colour cameras 28 a and 28 b and thedepth sensor 30 are arranged on a display unit support 36 of the headset 2 (which is discussed in more detail below) as shown in FIG. 4. Thecolour cameras 28 a, 28 b may be arranged over the eyes of the user suchthat they are spaced apart, for example, by the distance between thepupil axes of a user which is about 65 mm. Such an arrangement enables astereoscopic display to be captured and thus recreated in VR as will bediscussed in more detail in the following. The depth sensor 30 may bearranged between the two cameras 28 a, 28 b.

Referring to FIGS. 4a-4c the head set 2 comprises a display unit 32having a display means 34 a, 34 b for conveying visual information tothe user. In an advantageous embodiment the display means 34 comprises ahead-up display, which is mounted on an inner side of the display unitin front of the eyes of the user so that the user does not need toadjust their gaze to see the information displayed thereon. The head-updisplay may comprise a non-transparent screen, such an LCD or LED screenfor providing a full VR environment. Alternatively it may comprise atransparent screen, such that the user can see through the displaywhilst data is displayed on it. Such a display is advantageous inproviding an augmented reality AR. There may be two displays 34 a, 34 bone for each eye, or there may be a single display which is visible byboth eyes. The display unit may comprise a 2D or 3D display which may bea stereoscopic display. Although the system is described herein asproviding a VR image to a user, it will be appreciated that in otherembodiments the image mage be an augmented reality image, mixed realityimage or video image.

In the examples of FIGS. 4a-4c the display unit 32 is mounted to adisplay unit support 36. The display unit support 36 supports thedisplay unit 32 on the user and provides a removable support for thehead set 2 on the user. In the example the display unit support 36extends around the eyes of the user as best seen in FIGS. 4a to 4 c.

In an alternative embodiment the display unit 32 may be separate fromthe head set. For example the display means 34 comprises a monitor or TVdisplay screen or a projector and projector screen.

In an advantageous embodiment the system 10 comprises a head movementsensing unit 40. The head movement sensing unit comprises a movementsensing unit 42 for tracking head movement of a user as they move theirhead during operation of the system 10. The head movement sensing unit42 is configured to provide data in relation to the X, Y, Z coordinatelocation and the roll, pitch and yaw of a head of a user. This data isprovided to a head tracking module, which is discussed in more detail inthe following, and processes the data such that the display unit 32 canupdate the displayed VR images in accordance with head movement. Forexample, as the user moves their head to look to the left the displayedVR images move to the left. Whilst such an operation is not essential itis advantageous in providing a more immersive VR environment. In orderto maintain realism it has been found that the maximum latency of theloop defined by movement sensed by the head movement sensing unit 42 andthe updated VR image is 20 ms.

In an advantageous embodiment the head movement sensing unit 42comprises an acceleration sensing means 44, such as an accelerometerconfigured to measure acceleration of the head. In an advantageousembodiment the sensor 44 comprises three in-plane accelerometers,wherein each in-plane accelerometer is arranged to be sensitive toacceleration along a separate perpendicular plate. In this way thesensor is operable to measure acceleration in three-dimensions. However,it will be appreciated that other accelerometer arrangements arepossible, for example, there may only be two in-plane accelerometersarranged to be sensitive to acceleration along separate perpendicularplates such that two-dimensional acceleration is measured. Suitableaccelerometers include piezoelectric, piezoresistive and capacitivevariants.

In an advantageous embodiment the head movement sensing unit 42 furthercomprises a head orientation sensing means 47 which is operable toprovide data in relation to the orientation of the head. Examples ofsuitable head orientation sensing means include a gyroscope and amagnetometer 48 which are configured to measure the orientation of ahead of a user.

In an advantageous embodiment the head movement sensing unit 42 may bearranged on the head set 2. For example, the movement sensing unit 42may be housed in a movement sensing unit support 50 that is formedintegrally with or is attached to the display unit support 36.

In an advantageous embodiment the system 10 may comprise an eye gazesensing unit 100. The eye gaze sensing unit 100 comprises one or moreeye gaze sensors 102 for sensing the direction of gaze of the user. Inan advantageous embodiment the eye gaze sensor 102 comprises one or morecameras arranged in operation proximity to one or both eyes of the user.The or each camera 102 may be configured to track eye gaze by using thecentre of the pupil and infrared/near-infrared non-collimated light tocreate corneal reflections (CR). However, it will be appreciated thatother sensing means may be used for example: electrooculogram (EOG); oreye attached tracking. The data from the movement sensing unit 42 isprovided to an eye tracking module, which is discussed in more detail inthe following, and processes the data such that the display unit 32 canupdate the displayed VR images in accordance with eye movement. Forexample, as the user moves their eyes to look to the left the displayedVR images pan to the left. Whilst such an operation is not essential itis advantageous in providing a more immersive VR environment. In orderto maintain realism it has been found that the maximum latency of theloop defined by movement sensed by the eye gaze sensing unit 100 and theupdated VR image is about 50 ms, however in an advantageous embodimentit is 20 ms or lower.

In an advantageous embodiment the eye gaze sensing unit 100 may bearranged on the head set 2. For example, the eye gaze sensing unit 42may be attached to the display unit support 36 as shown in FIG. 4 a.

The control system 12 processes data from the physiological parametersensing system 14 and the position/motion detection system 16, andoptionally one or both of the head movement sensing unit 40 and the eyegaze sensing module 100, together with operator input data supplied toan input unit, to generate a VR (or AR) data which is displayed by thedisplay unit 32. To perform such a function, in the advantageousembodiment shown in FIGS. 1 and 2, the control system 12 may beorganized into a number of modules, such as: a skeletal tracking module52; a physiological parameter processing module 54; a VR generationmodule 58; a head tracking module 58; and an eye gaze tracking module104 which are discussed in the following.

The skeletal tracking module 52 processes the sensory data from theposition/motion detection system 16 to obtain joint position/movementdata for the VR generation module 58. In an advantageous embodiment theskeletal tracking module 52, as shown in FIG. 3b , comprises acalibration unit 60, a data fusion unit 62 and a skeletal tracking unit64 the operations of which will now be discussed.

The sensors 26 of the position/motion detection system 16 provide datain relation to the position/movement of a whole or part of a skeletalstructure of a user to the data fusion unit 62. The data may alsocomprise information in relation to the environment, for example thesize and arrangement of the room the user is in. In the exemplaryembodiment, wherein the sensors 26 comprise a depth sensor 30 and acolour cameras 28 a, 28 b the data comprises colour and depth pixelinformation.

The data fusion unit 62 uses this data, and the calibration unit 62, togenerate a 3D point cloud comprising a 3D point model of an externalsurface of the user and environment. The calibration unit 62 comprisesdata in relation to the calibration parameters of the sensors 26 and adata matching algorithm. For example, the calibration parameters maycomprise data in relation to the deformation of the optical elements inthe cameras, colour calibration and hot and dark pixel discarding andinterpolation. The data matching algorithm may be operable to match thecolour image from cameras 28 a and 28 b to estimate a depth map which isreferenced with respect to a depth map generated from the depth sensor30. The generated 3D point cloud comprises an array of pixels with anestimated depth such that they can be represented in a three-dimensionalcoordinate system. The colour of the pixels is also estimated andretained.

The data fusion unit 62 supplies data comprising 3D point cloudinformation, with pixel colour information, together with colour imagesto the skeletal tracking unit 64. The skeletal tracking unit 64processes this data to calculate the position of the skeleton of theuser and therefrom estimate the 3D joint positions. In an advantageousembodiment, to achieve this operation, the skeletal tracking unit isorganized into several operational blocks: 1) segment the user from theenvironment using the 3D point cloud data and colour images; 2) detectthe head and body parts of the user from the colour images; 3) retrievea skeleton model of user from 3D point cloud data; 4) use inversekinematic algorithms together with the skeleton model to improve jointposition estimation. The skeletal tracking unit 64 outputs the jointposition data to the VR generation module 58 which is discussed in moredetail in the following. The joint position data is time stamped by aclock module such that the motion of a body part can be calculated byprocessing the joint position data over a given time period.

Referring to FIGS. 2 and 3, the physiological parameter processingmodule 54 processes the sensory data from the physiological parametersensing system 14 to provide data which is used by the VR generationmodule 58. The processed data may, for example, comprise information inrelation to the intent of a user to move a particular body part or acognitive state of a user (for example, the cognitive state in responseto moving a particular body part or the perceived motion of a bodypart). The processed data can be used to track the progress of a user,for example as part of a neural rehabilitation program and/or to providereal-time feedback to the user for enhanced adaptive treatment andrecovery, as is discussed in more detail in the following.

The cortical activity is measured and recorded as the user performsspecific body part movements/intended movements, which are instructed inthe VR environment. Examples of such instructed movements are providedin the appended examples. To measure the cortical activity, the EEGsensors 22 are used to extract event related electrical potentials andevent related spectral perturbations, in response to the executionand/or observation of the movements/intended movements which can beviewed in VR as an avatar of the user.

For example the following bands provide data in relation to variousoperations: slow cortical potentials (SCPs), which are in the range of0.1-1.5 Hz and occur in motor areas of the brain provide data inrelation to preparation for movement; mu-rhythm (8-12 Hz) in the sensorymotor areas of the brain provide data in relation to the execution,observation and imagination of movement of a body part; betaoscillations (13-30 Hz) provide data in relation to sensory motorintegration and movement preparation. It will be appreciated that one ormore of the above potentials or other suitable potentials may bemonitored. Monitoring such potentials over a period of time can be usedto provide information in relation to the recovery or a user.

Referring to FIG. 5, an exemplary arrangement of sensors 20 is providedwhich is suitable for measuring neural events as a user performs varioussensorimotor and/or cognitive tasks. EEG sensors 22 may advantageouslybe arranged into groups to measure motor areas in one or more areas ofthe brain, for example: central (C1-C6, Cz); fronto-central (FC1-FC4,FCZ); centro-pariental (CP3, CP4, CPZ). In an advantageous embodimentcontra lateral EEG sensors C1, C2, C3 and C4 are arranged to measurearm/hand movements. The central, fronto-central and centro-parientalsensors may be used for measuring SCPs. EOG sensors 25 may further beprovided to measure eye movement signals. In this way the eye movementsignals can be isolated and accounted for when processing the signals ofother groups to avoid contamination.

In an advantageous embodiment the physiological parameter processingmodule 54 comprises a re-referencing unit 66 which is arranged toreceive data from the physiological parameter sensing system 14 andconfigured to process the data to reduce the effect of external noise onthe data. For example, it may process data from one or more of the EEG,EOG or EMG sensors. The re-referencing unit 66 may comprise one or morere-referencing blocks: examples of suitable re-referencing blocksinclude mastoid electrode average reference, and common averagereference. In the example embodiment a mastoid electrode averagereference is applied to some of the sensors and common average referenceis applied to all of the sensors. However, it will be appreciated thatother suitable noise filtering techniques may be applied to varioussensors and sensor groups.

In an advantageous embodiment, the processed data of the re-referencingunit 66 may be output to a filtering unit 68, however in an embodimentwithout re-referencing unit, the data from the physiological parametersensing system 14 is fed directly to the filtering unit 68. Thefiltering unit 68 may comprise a spectral filtering module 70 which isconfigured to band pass filter the data for one or more of the EEG, EOGand EMG sensors. In respect of the EEG sensors, the data may be bandpass filtered for one or more of the sensors to obtain the activity onone or more of the bands: SCPs, theta, alpha, beta, gamma, mu, gamma,and delta. In an advantageous embodiment the bands SCPs (0.1-1.5 Hz),alpha and mu (8-12 Hz), beta (18-30 Hz) delta (1.5-3.5 Hz), theta (3-8Hz), low gamma (30-100 Hz) and high gamma (above 100 Hz) are filteredfor all of the EEG sensors. In respect of EMG and EOG sensors similarspectral filtering may be applied but with different spectral filteringparameters. For example, for EMG sensors spectral filtering of a 30 Hzhigh pass cut off may be applied.

The filtering unit 66 may alternatively or additionally comprise aspatial filtering module 72. In an advantageous embodiment a spatialfiltering module 72 is applied to the SCPs band data from the EEGsensors (which is extracted by the spectral filtering module 70),however it may also be applied to other extracted bands. A suitable formof spatial filtering is spatial smoothing which comprises weightedaveraging of neighboring electrodes to reduce spatial variability of thedata. Spatial filtering may also be applied to data from the EOG and EMGsensors.

The filtering unit 66 may alternatively or additionally comprise aLaplacian filtering module 74, which is generally for data from the EEGsensors but may also be applied to data from the EOG and EMG sensors. Inan advantageous embodiment a Laplacian filtering module 72 is applied toeach of the Alpha, Mu and Beta band data of the EEG sensors which isextracted by the spectral filtering module 70, however it may be appliedto other bands. The Laplacian filtering module 72 is configured tofurther reduce noise and increase spatial resolution of the data.

The physiological parameter sensing system 14 may further comprise anevent marking unit 76. In an advantageous embodiment, when thephysiological parameter sensing system 14 comprises a re-referencingunit and/or a filtering unit 68, the event marking unit 76 is arrangedto receive processed data from either or both of these units whenarranged in series (as shown in the embodiment of FIG. 3c ). The eventmarking unit 76 is operable to use event based makers determined by anexercise logic unit (which will be discussed in more detail in thefollowing) to extract segments of sensory data. For example, when aspecific instruction to move a body part is sent to the user from theexercise logic unit, a segment of data is extracted within a suitabletime frame following the instruction. The data may, in the example of anEEG sensor, comprise data from a particular cortical area to therebymeasure the response of the user to the instruction. For example, aninstruction may be sent to the user to move their arm and the extracteddata segment may comprise the cortical activity for a period of 2seconds following instruction. Other example events may comprise:potentials in response to infrequent stimuli in the central andcentro-parietal electrodes; movement related potentials that are centralSCPs (slow cortical potentials) which appear slightly prior to movementand; error related potentials.

In an advantageous embodiment the event marking unit is configured toperform one or more of following operations: extract event relatedpotential data segments from the SCP band data; extract event relatedspectral perturbation marker data segments from Alpha and Beta or Mu orgamma band data; extract spontaneous data segments from Beta band data.In the aforementioned, spontaneous data segments correspond to EEGsegments without an event marker, and are different to event relatedpotentials, the extraction of which depends on the temporal location ofthe event marker.

The physiological parameter sensing system 14 may further comprise anartefact detection unit 78 which is arranged to receive the extracteddata segments from the event marking unit 76 and is operable to furtherprocess the data segments to identify specific artefacts in thesegments. For example, the identified artefacts may comprise 1) movementartefacts: the effect of a user movement on a sensor/sensor group 2)electrical interference artefacts: interference, typically 50 Hz fromthe mains electrical supply 3) eye movement artefacts: such artefactscan be identified by the EOG sensors 25 of the physiological parametersensing system 14. In an advantageous embodiment the artefact detectionunit 78 comprises an artefact detector module 80 which is configured todetect specific artefacts in the data segments. For example, anerroneous segment which requires deleting or a portion of the segmentwhich is erroneous and requires removing from the segment. Theadvantageous embodiment further comprises an artefact removal module 82,which is arranged to receive the data segments from the event markingunit 76 and artefact detected from the artefact detector module 80 toperform an operation of removing the detected artefact from the datasegment. Such an operation may comprise a statistical method such as aregression model which is operable to remove the artefact from the datasegment without loss of the segment. The resulting data segment isthereafter output to the VR generation module 58, wherein it may beprocessed to provide real-time VR feedback which may be based onmovement intention as will be discussed in the following. The data mayalso be stored to enable the progress of a user to be tracked.

In embodiments comprising other sensors, such as ECG, respirationsensors and GSR sensors, it will be appreciated that the data from suchsensors can be processed using one of more of the abovementionedtechniques where applicable, for example: noise reduction; filtering;event marking to extract event relate data segments; artefact removalfrom extracted data segments.

The head tracking module 56 is configured to process the data from thehead movement sensing unit 40 to determine the degree of head movement.The processed data is sent to the VR generation module 58, wherein it isprocessed to provide real-time VR feedback to recreate the associatedhead movement in the VR environment. For example, as the user movestheir head to look to the left the displayed VR images move to the left.

The eye gaze tracking module 104 is configured to process the data fromthe eye gaze sensing unit 100 to determine a change in gaze of the user.The processed data is sent to the VR generation module 58, wherein it isprocessed to provide real-time VR feedback to recreate the change ingaze in the VR environment.

Referring now to FIG. 3b , the VR generation module 58 is arranged toreceive data from the skeletal tracking module 52, physiologicalparameter processing module 54, and optionally one or both of the headtracking module 56 and the eye gaze tracking module 104, and isconfigured to process this data such that it is contextualized withrespect to a status of an exercise logic unit (which is discussed inmore detail in the following), and to generate a VR environment based onthe processed data.

The VR generation module may be organized into several units: anexercise logic unit 84; a VR environment unit 86; a body model unit 88;an avatar posture generation unit 90; a VR content integration unit 92;an audio generation unit 94; and a feedback generation unit 96.

The exercise logic unit 84 is operable to interface with a user input,such as a keyboard or other suitable input device. The user input may beused to select a particular task from a library of tasks and/or setparticular parameters for a task.

A body model unit 88 may be arranged to receive data from the exerciselogic unit 84 in relation to the particular part of the body requiredfor the selected task. For example this may comprise the entire skeletalstructure of the body or a particular part of the body such as an arm.The body model unit 88 thereafter retrieves a model of the required bodypart, for example from a library of body parts. The model may comprise a3D point cloud model, or other suitable model.

The avatar posture generation unit 90 is configured to generate anavatar based on the model of the body part from the body part model 88.

In an advantageous embodiment the VR environment unit 86 is arranged toreceive data from the exercise logic unit 84 in relation to theparticular objects which are required for the selected task. For examplethe objects may comprise a disk or ball to be displayed to the user.

The VR content integration unit may be arranged to receive the avatardata from the avatar posture generation unit 90 and the environment datafrom the VR environment unit 86 and to integrate the data in a VRenvironment. The integrated data is thereafter transferred to theexercise logic unit 58 and also output to the feedback generation unit86. The feedback generation unit 86 is arranged to output the VRenvironment data to the display means 34 of the head set 2.

During operation of the task the exercise logic unit 84 receives datacomprising joint position information from the skeletal tracking module64, data comprising physiological data segments from the physiologicalparameter processing module 54 data from the body model unit 88 and datafrom the VR environment unit 86. The exercise logic unit 84 is operableto processes the joint position information data which is in turn sentto the avatar posture generation unit 90 for further processing andsubsequent display. The exercise logic unit 84 may optionallymanipulated the data so that it may be used to provide VR feedback tothe user. Examples of such processing and manipulation includeamplification of erroneous movement; auto correction of movement toinduce positive reinforcement; mapping of movements of one limb toanother.

As the user moves, interactions and/or collisions with the objects, asdefined by the VR environment unit 86, in the VR environment, aredetected by the exercise logic unit 84 to further update the feedbackprovided to the user.

The exercise logic unit 84 may also provide audio feedback. For example,an audio generation unit (not shown) may receive audio data from theexercise logic unit, which is subsequently processed by the feedbackunit 94 and output to the user, for example, by headphones (not shown)mounted to the head set 2. The audio data may be synchronised with thevisual feedback, for example, to better indicate collisions with objectsin the VR environment and to provide a more immersive VR environment.

In an advantageous embodiment the exercise logic unit 84 may sendinstructions to the physiological parameter sensing system 14 to providefeedback to the user via one or more of the sensors 20 of thephysiological parameter sensing system 14. For example, the EEG 22and/or EMG 24 sensors may be supplied with an electrical potential thatis transferred to the user. With reference to the appended example, suchfeedback may be provided during the task. For example at stage 5,wherein there is no arm movement an electrical potential may be sent toEMG 24 sensors arranged on the arm and/or EEG sensors to attempt tostimulate the user into moving their arm. In another example, suchfeedback may be provided before initiation of the task, for instance, aset period of time before the task, to attempt to enhance a state ofmemory and learning.

In an advantageous embodiment the control system comprises a clockmodule 106. The clock module is used to assign time information to dataand various stages of input and output and processing. The timeinformation can be used to ensure the data is processed correctly, forexample, data from various sensors is combined at the correct timeintervals. This is particularly advantageous to ensure accuratereal-time processing of multimodal inputs from the various sensors andto generate real-time feedback to the user. The clock module may beconfigured to interface with one or more modules of the control systemto time stamp data. For example: the clock module 106 interfaces withthe skeletal tracking module 52 to time stamp data received from theposition/motion detection system 16; the clock module 106 interfaceswith the physiological parameter processing module 54 to time stamp datareceived from the physiological parameter sensing system 14; the clockmodule 106 interfaces with the head tracking module 58 to time stampdata received from the head movement sensing unit 40; the clock module106 interfaces with the eye gaze tracking module 104 to time stamp datareceived from the eye gaze sensing unit 100. Various operations on theVR generation module 58 may also interface with the clock module to timestamp data, for example data output to the display means 34.

Unlike complex conventional systems that connect several independentdevices together, in embodiments of the present inventionsynchronization occurs at the source of the data generation (for bothsensing and stimulation), thereby ensuring accurate synchronization withminimal latency and, importantly, low jitter. For example, for a stereohead-mounted display with refresh rate of 60 Hz, the delay would be assmall as 16.7 ms. This is not presently possible with a combination ofconventional stand-alone or independent systems. An important feature ofthe present invention is that it is able to combine a heterogeneousensemble of data, synchronizing them into a dedicated systemarchitecture at source for ensuring multimodal feedback with minimallatencies. The wearable compact head-mounted device allows easyrecording of physiological data from brain and other body parts.

Synchronization Concept:

Latency or Delay (T): It is the time difference between the moment ofusers actual action or brain state to the moment of its correspondingfeedback/stimulation. It is a positive constant in a typicalapplication. Jitter (ΔT) is the trial to trial deviation in Latency orDelay. For applications that require for instance immersive VR or AR,both latency T and jitter ΔT should be minimized to the least possible.Whereas in brain computer interface and offline applications, latency Tcan be compromised but jitter ΔT should be as small as possible.

Referring to FIGS. 1a and 1b , two conventional prior-art systemarchitectures are schematically illustrated. In these thesynchronization may be ensured to some degree but jitter (ΔT) is notfully minimized.

Design-I (FIG. 1a ):

In this design, the moment at which a visual cue is supplied to user isregistered directly in the computer while acquiring the EEG signal thatis acquired via a USB connection or serial connection. Meaning, thecomputer assumes, the moment at which it is registered with acquiredfrom user's brain is the moment a cue is displayed to the user. Notethat there are inherent delays and jitters in this design. First due tothe USB/serial port connectivity to computer, the registration of thesample into computer is has nonzero variable latency. Second, the momentthe display command is released from the computer, it undergoes variousdelay due to underlying display driver, graphical processing unit andsignal propagation, which is also not a constant. Hence these two kindsof delays add up and compromise alignment of visually evoked potentials.

Design-II (FIG. 1b ):

To avoid the above problem, it is known to use a photo diode to measurethe cue and synchronize its signal directly with an EEG amplifier. Inthis design, usually a photo-diode is placed on the display to sense alight. Usually, a cue is presented to user at the same time a portion ofscreen where the photo-diode is attached is lighted up. This way themoment at which the cue is presented is registered with photo-diode andsupplied to EEG amplifier. This way EEG and visual cue information aredirectly synchronized at source. This procedure is accurate foralighting visually evoked trials, however has a number of drawbacks:

-   -   the number of visual cues it can code are limited to number of        photodiodes. A typical virtual reality based visual stimulation        would have large number of events to be registered together with        physiological signals accurately.    -   the use of photo-diode in a typical micro-display (e.g., 1        square inch size, with pixel density of 800×600) of a        head-mounted display would be difficult and even worse reduces        usability. Note also that for the photo-diode to function, ample        light should be supplied to the diode resulting in a limitation.    -   the above drawbacks are further complicated, when a plurality of        stimuli (such as audio, magnetic, electrical and mechanical are        needed to synchronize with plurality of sensors data (such as        EEG, EMG, ECG, video camera, inertial sensors, respiration        sensor, pulse oximetry, galvanic skin potentials etc.).

In embodiments of the present invention, the above drawbacks areaddressed to provide a system that is accurate and scalable to manydifferent sensors and many different stimuli. This is achieved byemploying a centralized clock system that supplies a time-stampinformation and each sensor's samples are registered in relation to thisto the time-stamp.

In an embodiment, each stimulation device may advantageously be equippedwith an embedded sensor whose signal is registered by a synchronizationdevice. This way, a controller can interpret plurality of sensor dataand stimulation data can be interpreted accurately for further operationof the system.

In an embodiment, in order to reduce the amount of data to synchronizefrom each sensor, instead of using a real sensor, video content codefrom a display register may be read.

Referring to FIG. 2a , an embodiment of the invention in which thecontent fed to a micro-display on the headset is synchronized with brainactivity signals (EEG signals) is schematically illustrated.

Generally, the visual/video content that is generated in the controlsystem is first pushed to a display register (a final stage before thevideo content is activated on the display). In our design together withvideo content, the controller sends a code to a part of the register(say N bits) corresponding to one or more pixels (not too many pixels,so that the user is not disturbed; the corner pixels in the microdisplay are recommended as they may not be visible to user). The codewill be defined by controller describing what exactly the displaycontent is. Now using a clock signal the acquisition module reads thecode from the display register and attaches a time stamp and sends tonext modules. At the same moment EEG samples are also sampled andattached with the same time stamp. This way when EEG samples and thevideo code samples are arrived at the controller, these samples could beinterpreted accordingly.

Note that all these modules are employed in one embedded system that hasa single clock. This leads least latency as well as least jitter.

The same principle may be used for an audio stimulation as illustratedin FIG. 2b . The audio stimulation can be sampled by the data sent to adigital to analog (DAC) converter.

More generally, any kind of stimulation, as illustrated in FIG. 2c ,(such as trans-cranial stimulations (tACS), tDCS, TMS, etc.) could bedirected to the acquisition module using a sensor and an analog todigital (ADC) converter. This can also be achieved by sending thedigital signals supplied to DAC as illustrated in the case of audiostimulation. Plural data from an EEG, video camera data or any othersensor (e.g. INS: Inertial sensor) is synchronized in the sameframework. Note that each sensor or stimulation could be sampled withdifferent sampling frequency. An important point is that the sensor orstimulation data samples are attached with the time-stamp defined withthe clock module.

Example 1: Operation of System (10) in Exemplary “Reach an Object” Task

In this particular example an object 110, such as a 3D disk, isdisplayed in a VR environment 112 to a user. The user is instructed toreach to the object using a virtual arm 114 of the user. In the firstinstance the arm 114 is animated based on data from the skeletaltracking module 16 derived from the sensors of the position/motiondetection system 16. In the second instance, wherein there is negligibleor no movement detected by the skeletal tracking module 16, then themovement is based data relating to intended movement from thephysiological parameter processing module 52 detected by thephysiological parameter sensing system 14, and in particular the datamay be from the EEG sensors 22 and/or EMG sensors 24.

FIGS. 7 and 8 a-8 g describe the process in more detail. At stage 1 inFIG. 7, a user, such as a patient or operator, interfaces with a userinput of the exercise logic unit 84 of the VR generation module 58 toselect a task from a library of tasks which may be stored. In thisexample a ‘reach an object task’ is selected. At this stage the user maybe provided with the results 108 of previous like tasks, as shown inFIG. 8a . These results may be provided to aid in the selection of theparticular task or task difficulty. The user may also input parametersto adjust the difficulty of the task, for example based on a level ofsuccess from the previous task.

At stage 2, the exercise logic unit 84 initializes the task. Thiscomprises steps of the exercise logic unit 84 interfacing with the VRenvironment unit 86 to retrieve the parts (such as the disk 110)associated with the selected task from a library of parts. The exerciselogic unit 84 also interfaces with the body model unit 88 to retrieve,from a library of body parts, a 3D point cloud model of the body part(in this example a single arm 114) associated with the exercise. Thebody part data is then supplied to the avatar posture generation unit 90so that an avatar of the body part 114 can be created. The VR contentintegration unit 92 receives data in relation to the avatar of the bodypart and parts in the VR environment and integrates them in a VRenvironment. This data is thereafter received by the exercise logic unit84 and is output to the display means 34 of the head set 2 as shown inFIG. 8b . The target path 118 for the user to move a hand 115 of the arm114 along is indicated, for example, by colouring it blue.

At stage 3, the exercise logic unit 84 interrogates the skeletaltracking module 16 to determine whether any arm movement has occurred.The arm movement being derived from the sensors of the position/motiondetection system 16 which are worn by the user. If a negligible amountof movement (for example an amount less than a predetermined amount,which may be determined by the state of the user and location ofmovement) or no movement has occurred then stage 5 is executed, elsestage 4 is executed.

At stage 4 the exercise logic unit 84 processes the movement data todetermine whether the movement is correct. If the user has moved theirhand 115 in the correct direction, for example, towards the object 110,along the target path 118, then stage 4 a is executed and the colour ofthe target path may change, for example it is coloured green, as shownin FIG. 8c . Else, if the user moves their hand 115 in an incorrectdirection, for example, away from the object 110, Then stage 4 b isexecuted and the colour of the target path may change, for example it iscoloured red, as shown as FIG. 8 d.

Following stage 4 a and 4 b stage 4 c is executed, wherein the exerciselogic unit 84 determines whether the hand 115 has reached the object110. If the hand has reached the object, as shown in FIG. 8e then stage6 is executed, else stage 3 is re-executed.

At stage 5 the exercise logic unit 84 interrogates the physiologicalparameter processing module 52 to determine whether any physiologicalactivity has occurred. The physiological activity is derived from thesensors of the physiological parameter sensing system module 14, whichare worn by the user, for example the EEG and/or EMG sensors. EEG andEMG sensors may be combined to improve detection rates, and in theabsence of a signal from one type of sensor a signal from the other typeof sensor maybe used. If there is such activity, then it may beprocessed by the exercise logic unit 84 and correlated to a movement ofthe hand 115. For example a characteristic of the event related datasegment from the physiological parameter processing module 52, such asthe intensity or duration of part of the signal, may be used tocalculate a magnitude of the hand movement 115. Thereafter stage 6 isexecuted.

At stage 6 a if the user has successfully completed the task, then toprovide feedback 116 to the user a reward score may be calculated, whichmay be based on the accuracy of the calculated trajectory of the hand115 movement. FIG. 8e shows the feedback 116 displayed to the user. Theresults from the previous task may also be updated.

Thereafter stage 6 b is executed, wherein a marker strength of thesensors of the physiological parameter sensing system module 14, forexample the EEG and EMG, sensors may be used to provide feedback 118.FIG. 8f shows an example of the feedback 120 displayed to the user,wherein the marker strength is displayed as a percentage of a maximumvalue. The results from the previous task may also be updated.Thereafter, stage 7 is executed, wherein the task is terminated.

As stage 8 if there is no data provided by either of the sensors of thephysiological parameter sensing system module 14 or the sensors of theposition/motion detection system 16 with in a set period of time thentime out 122 occurs, as shown in FIG. 8g and stage 7 is executed.

Example 2: Hybrid Brain-Computer Interface with Virtual Reality Feedbackwith Head-Mounted Display, Robotic System and Functional ElectricalStimulation

Objective:

To provide optimal training for patients with upper movements movementdeficits resulting from neurological problems (e.g., ALS, stroke, braininjury, locked-in syndrome, Parkinson disease etc.). These patientswould require training to reintegrate the lost/degraded movementfunction. A system that reads their intention to make a functionalmovement and provide an assistance in completing the movement couldenhance the rehabilitation outcome.

For this purpose, the system could exploit Hebbian learning inassociating brain's input and output areas in reintegrating the lostmovement function. The Hebbian principle is “Any two systems of cells inthe brain that are repeatedly active at the same time will tend tobecome ‘associated’, so that activity in one facilitates activity in theother.”

In the present example, the two systems of cells are the areas of thebrain that are involved in sensory processing and in generating motorcommand. When the association is lost due to neural injury, it could berestored or re-built via Hebbian training. For the optimal results ofthis training, one must ensure near perfect synchronization of systeminputs and outputs and in providing realtime multi-sensory feedback tothe patient with small delay and more importantly almost negligiblejitter.

The physical embodiment illustrated in FIG. 9, comprises a wearablesystem having a head-mounted display (HMD) 19 to display virtual reality3D video content on micro-displays (e.g., in first person perspective),a stereo video camera 30 and a depth camera 28, whose data is used fortracking the wearers own arm, objects and any second person under thefield of view (motion tracking unit). Additionally, the EEG electrodes22 placed over the head of the wearer 1, integrated in a head set 2 aspreviously described (not shown in detail in this figure forsimplification), EMG electrodes 24 placed on the arm will measureelectrical activity of the brain and of muscles respectively, used forinferring user's intention in making a goal directed movement.Additionally, there exists an Inertial Measurement Unit (IMU) 29 that isused for tracking head movements. The executed or intended movements arerendered in the virtual reality display. In case of evidence of themovements through the biological sensor data (ie, EEG, EMG, and motiontracing) feedback mechanisms aid the patient in making goal directedmovement using a robotic system 41. Furthermore, functional electricalstimulation (FES) system 31 activates muscles of the arm in completingthe planned movement. Additionally, the feedback mechanisms shallprovide appropriate stimulation tightly coupling to the intention tomove to ensure the implementation of Hebbian learning mechanism.

The following paragraph describes a typical trial in performing atypical goal directed task, which could be repeated by the patientseveral times to complete a typical training session. As shown in FIG.10, a 3D visual cue 81, in this case a door knob, when displayed in theHMD could instruct the patient 1 to make a movement corresponding toopening the door. Followed by the visual cue, the patient may attempt tomake the suggested movement. Sensor data (EEG, EMG, IMU, motion data) isacquired in synchronization with the moment of presentation of thevisual cue. The control system 51 then extracts the sensor data andinfers user intention and a consensus is made in providing feedback tothe user through a robot 41 that moves the arm, and HMD displaysmovement of an avatar 83, which is animated based on the inferred data.A Functional Electrical Stimulation (FES) 31 is also synchronizedtogether with other feedbacks ensuring a congruence among them.

An exemplary architecture of this system is illustrated in FIG. 2d . Theacquisition unit acquires physiological data (i.e., EEG 22, EMG 24, IMU29 and camera system 30). The camera system data include stereo videoframes and depth sensor data. Additionally the stimulation related datasuch as the moment at which a particular image frame of the video isdisplayed on the HMD, robot's motor data and sensors 23 and that of FES31 stimulation data are also sampled by the acquisition unit 53. Thisunit associates each sensor and stimulation sample with a time stamp(TS) obtained from the clock input. The synchronized data is thenprocessed by control system and is used in generating appropriatefeedback content to the user through VR HMD display, robotic movement aswell as FES stimulation.

Inputs of the System:

-   -   Inertial measurement unit (IMU) sensors 29, for instance        including an accelerometer, a gyroscope, a magneto-meter:        Purpose, to track head movements. This data is used for        rendering VR content as well as to segment EEG data where the        data quality might be degraded due to movement.    -   Camera system 30, 28: The camera system comprises a stereo        camera 30, and a depth sensor 28. The data of these two sensors        are combined to compute tracking data of a wearer's own        movements of upper limbs, and for tracking wearer's own arm        movements. These movements are then used in animating the avatar        in the virtual reality on micro displays 32 and in detecting if        there was a goal directed movements, which is then used for        triggering feedback through display 32, robot 41, and        stimulation device FES 31. In an exemplary application of the        invention the sensors EEG 22 & EMG 24 may be used for inferring        if there was an intention to make a goal directed movement.

Outputs of the System/Feedback Systems

-   -   Micro-displays 34 of head set 2: Renders 2D/3D virtual reality        content, where a wearer experiences the first person perspective        of the virtual world as well as of his own avatar with its arms        moving in relation to his own movements.    -   Robotic system 41: Robotic system described in this invention is        used for driving movements of the arm, where the user 1 holds a        haptic knob. The system provides a range of movements as well as        haptic feedback of natural movements of activities of daily        living.    -   Functional Electrical Stimulation (FES) device 31: Adhesive        electrodes of FES system are placed on user's arms to stimulate        nerves, which up on activated can restore the lost voluntary        movements of the arm. Additionally, the resulting movements of        the hand results in kinesthetic feedback to the brain.

Data Processing

The following paragraphs describe the data manipulations from inputstill outputs.

Acquisition Unit 53:

The description of acquisition unit 53 ensures near perfectsynchronization of inputs/sensor data and outputs/Stimulation/feedbackof the system as illustrated in the FIG. 11. Each sensor data may havedifferent sampling frequency and whose sampling may have not initiatedat exact same moment due to non-shared internal clock. In this example,the sampling frequency of EEG data is 1 kHz, EMG data is 10 KHz, IMUdata is 300 Hz, Video camera data is 120 frames per second (fps).Similarly, the stimulation signals have different frequencies, where thedisplay refresh rate is at 60 Hz, robot sensors of 1 KHz, and FES dataat 1 KHz.

The acquisition unit 53 aims at solving the issue of synchronization ofinputs and outputs accurately. In achieving so, the outputs of thesystem are sensed either with dedicated sensors or indirectly recordedfrom a stage before stimulation, for instance as follows:

-   -   Sensing the micro-display: Generally, the video content that is        generated in the control system is first pushed to a display        register 35 (a final stage before the video content is activated        on the display). Together with video content, the controller        sends a code to a part of the register (say N bits)        corresponding to one or more pixels (not too many pixels, so        that the user is not disturbed). The corner pixels in the micro        display are preferred as they may not be visible to user. The        codes (a total of 2̂N) may be defined by the controller or the        exercise logic unit describing the display content.    -   Sensing FES: The FES data can be red from its last stage of        generation, i.e., from the DAC.    -   Sensing Robot's movements: The robots motors are embedded with        sensors providing information on angular displacement, torque        and other control parameters of the motors.

Now using a clock signal with preferably a much higher frequency thanthat of the inputs and outputs (e.g., 1 GHz), but at least double thehighest sampling frequency among sensors and stimulation units, theacquisition module reads the sensor samples and attaches a time stamp asillustrated in the FIG. 12. When a sample of a sensor arrives from itsADC 37 a, its time of arrival is annotated with next immediate risingedge of the clock signal. Similarly for every sensor and stimulationdata a time-stamp is associated. When these samples arrive at thecontroller, it interprets the samples according to the time stamp ofarrival leading to minimized jitters across sensors and stimulations.

Physiological Data Analysis

The physiological data signals EEG and EMG are noisy electrical signalsand preferably are pre-processed using appropriate statistical methods.Additionally the noise can also be reduced by better synchronizing theevents of stimulation and behavior with the physiological datameasurements with negligible jitter.

FIG. 13 illustrates various stages of the pre-processing (filtering 68,epoch extraction and feature extraction stages). EEG samples from allthe electrodes are first spectrally filtered in various bands (e.g.,0.1-1 Hz, for slow cortical potentials, 8-12 Hz for alpha waves andRolandic mu rhythms, 18-30 Hz for beta band and from 30-100 Hz for gammaband). Each of these spectral bands contains different aspects of neuraloscillations at different locations. Following this stage the signalsundergo spatial filtering to improve signal-to-noise ratio additionally.The spatial filters include simple processes such as common averageremoval to spatial convolution with Gaussian window or Laplace windows.Following this stage the incoming samples are segmented into temporalwindows based on event markers arriving from event manager 71. Theseevents correspond to the moment the patient is given a stimulus or madea response.

These EEG segments are then fed to feature extraction unit 69, wheretemporal correction is first made. One simple example of temporalcorrection is removal of baseline or offset from the trial data from aselected spectral band data. The quality of these trials may be assessedusing statistical methods such as Outliers detection. Additionally, ifthere is a head movement registered through IMU sensor data, the trialsare annotated as artefact trials. Finally features are computed fromeach trial that well describe the underlying neural processing. Thesefeatures are then fed to a statistical unit 67.

Similarly, the EMG electrode samples are first spectrally filtered, andapplied a spatial filter. The movement information is obtained from theenvelope or power of the EMG signals. Similar to EEG trials, EMGspectral data is segmented and passed to feature extraction unit 69. Theoutput of EMG feature data is then sent to statistical unit 67.

The statistical unit 67 combines various physiological signals andmotion data to interpret the intention of the user in performing a goaldirected movement. This program unit includes mainly machine learningmethods for detection, classification and regression analysis ininterpretation of the features. The outputs of this module are intentionprobabilities and related parameters which drive the logic of theexercise in the Exercise logic unit 84. This exercise logic unit 84generates stimulation parameters which are then sent to afeedback/stimulation generation unit of the stimulation system 17.

Throughout these stages, it is ensured to have minimal lag and moreimportantly least jitter.

Event Detection & Event Manager

Events such as the moment at which the patient is stimulated orpresented an instruction in the VR display, the moment at which thepatient performed an action are necessary for the interpretation of thephysiological data. FIG. 14 illustrates event detection. The eventscorresponding to movements and those of external objects or of a secondperson need to be detected. For this purpose the data from camera system30 (stereo cameras, and 3D point cloud from the depth sensor) areintegrated in the tracking unit module 73 to produce various trackinginformation such as: (i) patient's skeletal tracking data, (ii) objecttracking data, and (iii) a second user tracking data. Based on therequirements of the behavioral analysis, these tracking data may be usedfor generating various events (e.g., the moment at which patient liftshis hand to hold door knob).

IMU data provides head movement information. This data is analyzed toget events such as user moving head to look at the virtual door knob.

The video display codes correspond to the video content (e.g., displayof virtual door knob, or any visual stimulation). These codes alsorepresent visual events. Similarly FES stimulation events, Robotmovement and haptic feedback events are detected and transferred intoevent manager 71. Analyzer modules 75, including a movement analyzer 75a, an IMU analyzer 75 b, an FES analyzer 75 c and a robot sensoranalyzer 75 d, process the various sensor and stimulation signals forthe event manager 71.

The event manager 71 then sends these events for tagging thephysiological data, motion tracking data etc. Additionally these eventsalso sent to Exercise logic unit for adapting the dynamics of exerciseor challenges for the patient.

Other Aspects of Control System

The control system interprets the incoming motion data, intentionprobabilities from the physiological data and activates exercise logicunit and generates stimulation/feedback parameters. The following blocksare main parts of the control system.

-   -   VR feedback: The motion data (skeletal tracking, object tracking        and user tracking data) is used for rendering 3D VR feedback on        the head-mounted displays, in form of avatars and virtual        objects.    -   Exercise logic unit 84: The exercise logic unit implements        sequence of visual display frames including instructions and        challenges (target task to perform, in various difficulty        levels) to the patient. The logic unit also reacts to the events        of the event manager 71. Finally this unit sends stimulation        parameters to the stimulation unit.    -   Robot & FES stimulation generation unit: this unit generates        inputs required to perform a targeted movement of the robotic        system 41 and associated haptic feedback. Additionally,        stimulation patterns (current intensity and electrode locations)        for the FES module could be made synchronous and congruent to        the patient.

Example 3: Brain Computer Interface and Motion Data Activated NeuralStimulation with Augmented Reality Feedback

Objective

A system could provide precise neural stimulation in relation to theactions performed by a patient in real world, resulting in reinforcementof neural patterns for intended behaviors.

Description

Actions of the user and that of a second person and objects in the sceneare captured with a camera system for behavioral analysis. Additionallyneural data is recorded with one of of the modalities (EEG, ECOG etc.)are synchronized with IMU data. The video captured from the camerasystem is interleaved with virtual objects to generate 3D augmentedreality feedback and provided to the user though head-mounted display.Finally, appropriate neural stimulation parameters are generated in thecontrol system and sent to the neural stimulation.

For delay and jitter between user's behavioral and physiologicalmeasures and neural stimulation should be optimized for effectivereinforcement of the neural patterns.

The implementation of this example is similar to Example 2, except thatthe head-mounted display (HMD) displays Augmented Reality contentinstead of Virtual Reality (see FIG. 2e ). Meaning, virtual objects areembedded in 3D seen captured using stereo camera and displayed on microdisplays insuring first person perspective of the scene. Additionally,direct neural stimulation in implemented through such as deep brainstimulation and cortical stimulation, and non-invasive stimulations suchas trans-cranial direct current stimulation (tDCS), trans-cranialalternating current stimulation (tACS), trans-cranial magneticstimulation (TMS) and trans-cranial Ultrasonic stimulation. The systemcan advantageously use one or more than one stimulation modalities attime to optimize the effect. This system exploits the acquisition unitdescribed in the example 1.

LIST OF REFERENCES

-   10 Physiological parameter measurement and motion tracking system-   2 Head set    -   3 EEG sensing device        -   4 EEG sensor support            -   4 a central branch            -   4 b front lateral branch                -   4 b 1, 4 b 2 back and side extensions            -   4 c center lateral branch                -   4 c 1, 4 c 2 front, rear extensions            -   4 d rear lateral branch                -   4 d 1, 4 d 2 front and side extensions            -   401 base wall                -   404 electrode orifices                -    404 a electrode orifice portion                -    404 b through passage portion                -   405 tensioner anchors-fixing orifices                -   406 bottom surface            -   402 side walls            -   403 channel            -   5 potting material (top wall)                -   top surface 506        -   6 Flexible circuit            -   22 EEG sensors                -   221 EEG electrodes            -   8 EEG signal processing circuit                -   8 a discrete EEG signal amplifiers                -   8 b circuit traces            -   601 flexible circuit substrate                -   602 electrode orifice                -   605 tensioner orifice            -   6 a central branch            -   6 b front lateral branch                -   6 b 1, 6 b 2 back and side extensions            -   6 c center lateral branch                -   6 c 1, 6 c 2 front, rear extensions            -   6 d rear lateral branch                -   6 d 1, 6 d 2 front and side extensions            -   41 connection portion        -   7 tensioners 4 elastic ties        -   37 conducting gel    -   9 head mount frame support        -   strap        -   adjustment knob    -   19 Head-mounted display        -   32 Display unit            -   34 Display means            -   35 Display register        -   36 Display unit support        -   33 Audio unit        -   100 Eye gaze sensing Unit            -   102 eye gaze sensor        -   40 Head movement sensing Unit            -   42 Movement sensing unit                -   44 Acceleration sensing means                -   47 Head orientation sensing means                -    46 Gyroscope                -   48 Magnetometer        -   50 movement sensing unit support (mount to HMD system)-   12 Control system    -   51 Control module        -   57 output signals (video, audio, stimulation)    -   53 Acquisition module    -   55 Memory    -   52 Skeletal tracking Module        -   60 Data fusion unit        -   62 Calibration unit        -   64 Skeletal tracking unit    -   54 Physiological parameter processing Module        -   66 Re-referencing unit        -   68 Filtering unit            -   70 Spectral filtering module            -   72 Spatial smoothing filtering module            -   74 Laplacian filtering module            -   76 Event marking unit            -   78 Artefact unit                -   80 Artefact detecting module                -   82 Artefact removal module            -   69 feature extraction unit            -   67 statistical unit        -   56 Head tracking module        -   104 Eye gaze tracking module        -   58 VR generation module            -   84 Exercise logic unit                -   Input unit            -   86 VR environment unit            -   88 Body model unit            -   90 Avatar posture generation unit            -   92 VR content integration unit            -   94 Audio generation unit            -   96 Feedback generation unit        -   106 Clock module        -   71 Events manager        -   73 Tracking unit            -   User tracking                -   →64 Skeletal tracking unit                -   →104 Eye gaze tracking module            -   Object tracking        -   75 Analyzer modules            -   75 a Movement            -   75 b IMU            -   75 c FES            -   75 d Robot sensor-   13 Sensing system    -   14 Physiological parameter sensing system        -   20 Sensors        -   22 Electroencephalogram (EEG) sensors        -   24 Electromyogram (EMG)—connected to muscles in body        -   25 Electrooculography (EOG)—eye movement sensor        -   27 Electrocardiogram (ECG)        -   29 Inertial Sensor (INS)/Inertial measurement unit (IMU)            sensor            -   40 Head movement sensing Unit        -   Body temperature sensor        -   Galvanic skin sensor    -   16 Position/motion detection system        -   26 Sensors            -   28 Depth/distance sensor            -   30 Camera (colour)        -   21 sensor output signals-   17 Stimulation system    -   31 Functional Electrical Stimulation (FES) system    -   Audio stimulation system→audio unit 33    -   Video stimulation system→display unit 32    -   37 a Analogue to Digital Converter (ADC)    -   37 b Digital to Analogue Converter (DAC)    -   39 content code signal    -   41 Haptic feedback device→robot        -   23 user feedback sensors

1. A head set comprising: a brain electrical activity (EEG) sensingdevice comprising EEG sensors configured to be mounted on a head of awearer so as to position the EEG sensors at selected positions ofinterest over the wearers scalp, the EEG sensing device comprising asensor support and a flexible circuit assembled to the sensor support,the sensor support and flexible circuit comprising a central stemconfigured to extend along a center plane of the top of a wearer's headin a direction from nasion to inion, a front lateral branch configuredto extend across a front portion of the wearer's head extendinglaterally from the central stem, a center lateral branch configured toextend across a top portion of the wearer's head essentially between thewearer's ears, and a rear lateral branch configured to extend across aback portion of the wearer's head, wherein the sensor support comprisesa base wall having a first side to contact the head of the wearer andside walls extending along edges of an opposite side of the base wall toform an essentially flat “U” shaped channel in which the flexiblecircuit is inserted and the base wall comprise EEG sensor orifices toreceive a gel to allow electrical contact to be established between thescalp of the wearer and contacts of EEG sensors on the flexible circuit;further comprising a head-mounted display (HMD) fixed to a head mountframe support, wherein said “U” shaped channel is configured to bend toconform to a generally spherical or ellipsoid three dimensional fromcorresponding essentially to the general morphology of a top half of ahuman head; and a head-mounted display (HMD) fixed to a head mount framesupport and configured to be positioned over the eyes of a wearer of theheadset, wherein the HMD comprises a display unit having a displaycomprising an electronic screen configured for positioning in front ofthe wearer's eyes to present visual information to the wearer, whereinsaid visual information is provided as part of a VR (virtual reality)environment or an AR (augmented reality) environment, the HMD furthercomprising a position/motion detection system operable to detect aposition/motion of a body part of a user, the position/motion/detectionsystem comprising one or more color cameras, and a depth sensor.
 2. Thehead set according to claim 1 wherein each of the lateral branchesfurther comprises extensions the extensions including extensionsextending in a front to rear or in a rear to front direction, andwherein EEG sensors are positioned in discrete spaced apart positionsalong the stem, branches and extensions.
 3. The head set according toclaim 1 wherein a top wall or flexible sealing material is mounted orfilled over the flexible circuit in the channel in order to seal in awaterproof manner the electrical circuit tracks and components on theflexible circuit within the channel.
 4. The head set according to claim1 wherein the sensor support is a single piece part molded or formedfrom a flexible polymeric material.
 5. The head set according to claim 1wherein the flexible circuit comprises a single piece flexiblesubstrate.
 6. The head set according to any preceding claim wherein thesensor support further comprises tensioner anchors configured to anchorelastic tensioners between positions in the stem, branches andextensions of the EEG sensing device, and also between the EEG sensingdevice and a head mount frame support wherein the flexible circuitcomprises orifices for the tensioning anchors.
 7. (canceled)
 8. The headset according to claim 1 wherein next to each EEG sensor on the flexiblecircuit is positioned a discrete EEG signal amplifier configured toamplify the brain electrical activity signal picked up by thecorresponding EEG sensor.
 9. The head set according to claim 1 whereinthe EEG sensors comprise electrodes in the form of conductive circuitpads on a surface of a substrate of the flexible circuit intended toface the wearer's scalp.
 10. The head set according to claim 1 whereinthe EEG sensors comprise protruding conductive compressible elementsmounted on the flexible substrate and electrically connected to acircuit trace of a substrate of the flexible circuit.
 11. (canceled) 12.The head set of claim 1, further comprising a head movement sensing unitat the HMD.
 13. The head set according to claim 1 further comprising awireless communication device to interconnect the HMD to externalelectronic devices and computing systems in a wireless fashion and anonboard power supply to power the HMD, located at the HMD. 14.(canceled)
 15. The head set according to claim 1, wherein the headsetfurther incorporates a plurality of sensors configured to measuredifferent physiological parameters, selected from a group consisting ofECOG sensors, eye movement sensors, and head movement sensing unit. 16.(canceled)
 17. The head set according to claim 11, wherein the flexiblecircuit comprises a pluggable electrical connector for plugging to acomplementary pluggable electrical connector on the HMD.
 18. The headset according to claim 1, wherein the flexible circuit comprisesorifices adjacent the EEG sensor contacts or electrodes, and wherein theEEG sensor orifices overlap the flexible circuit orifices such that athrough passage between a top surface and a bottom surface of thesensing device is provided.
 19. A head set comprising a brain electricalactivity (EEG) sensing device comprising EEG sensors configured to bemounted on a head of a wearer so as to position the EEG sensors atselected positions of interest over the wearers scalp, the EEG sensingdevice comprising a sensor support and a flexible circuit assembled tothe sensor support, the sensor support and flexible circuit comprising acentral stem configured to extend along a center plane of the top of thehead in a direction from nasion to inion, a front lateral branchconfigured to extend across a front portion of a wearer's head extendinglaterally from the central stem, a center lateral branch configured toextend across a top portion of a wearer's head essentially between thewearer's ears, and a rear lateral branch configured to extend across aback portion of a wearer's head, wherein the sensor support comprises abase wall comprises EEG sensor orifices to allow access to the EEGsensor contacts or electrodes on the flexible circuit, and wherein theflexible circuit comprises orifices adjacent the EEG sensor contacts orelectrodes overlapping the EEG sensor orifices such that a throughpassage between a top surface and a bottom surface of the sensing deviceis provided.
 20. A head set comprising a brain electrical activity (EEG)sensing device comprising EEG sensors configured to be mounted on a headof a wearer so as to position the EEG sensors at selected positions ofinterest over the wearers scalp, the EEG sensing device comprising asensor support and a flexible circuit assembled to the sensor support,the sensor support and flexible circuit comprising a central stemconfigured to extend along a center plane of the top of the head in adirection from nasion to inion, a front lateral branch configured toextend across a front portion of a wearer's head extending laterallyfrom the central stem, a center lateral branch configured to extendacross a top portion of a wearer's head essentially between the wearer'sears, and a rear lateral branch configured to extend across a backportion of a wearer's head, wherein each of the lateral branches furthercomprises extensions, the center lateral branches comprising extensionsextending both in a front to rear and in a rear to front direction suchthat the center lateral branches can be elastically tensioned to boththe front lateral branches and the rear lateral branches.
 21. (canceled)22. A physiological parameter measurement system comprising a head set,a control system, a sensing system; and a stimulation system, thesensing system comprising one or more physiological sensors including atleast brain electrical activity sensors mounted in the head set, the EEGsensors configured to be mounted on a head of a wearer at selectedpositions of interest over the wearers scalp, the headset comprising asensor support and a flexible circuit assembled to the sensor support,the sensor support and flexible circuit comprising a central stemconfigured to extend along a center plane of the top of the head in adirection from nasion to inion, a front lateral branch configured toextend across a front portion of a wearer's head extending laterallyfrom the central stem, a center lateral branch configured to extendacross a top portion of a wearer's head essentially between the wearer'sears, and a rear lateral branch configured to extend across a backportion of a wearer's head, the stimulation system comprising one ormore stimulation devices including at least a visual stimulation system,the control system comprising an acquisition module configured toreceive sensor signals from the sensing system, and a control moduleconfigured to process the signals from the acquisition module andcontrol the generation of stimulation signals to one or more devices ofthe stimulation system, wherein the control system further comprises aclock module and wherein the control system is configured to time stampsignals related to the stimulation signals and the sensor signals with aclock signal from the clock module, enabling the stimulation signals tobe synchronized with the sensor signals by means of the time stamps, andwherein said time stamped signals related to the stimulation signalscomprise content code signals received from the stimulation system;wherein the sensing system comprises physiological sensors selected froma group comprising Electromyogram (EMG) sensors, Electrooculography(EOG) sensors, Electrocardiogram (ECG) sensors, Inertial Sensors (INS),Body temperature sensor, Galvanic skin sensor, pulse oximetry sensor,respiration sensors; and wherein wherein the stimulation systemcomprises stimulation devices selected from a group comprising audiostimulation device, Functional Electrical Stimulation (FES) devices, andhaptic feedback devices, said functional electrical stimulation (FES)devices being connected to the control system and operable toelectrically stimulate one or more body parts of the user, the FESdevices selected from a group consisting of electrodes configured tostimulate nerves or muscles, trans-cranial alternating currentstimulation (tACS), direct current stimulation (tDCS), trans-cranialmagnetic stimulation (T S) and trans-cranial ultrasonic stimulation. 23.The system according to claim 22 further comprising a display registerconfigured to receive display content representing a final stage beforethe display content is activated on the display, the display registerbeing configured to generate a display content code signal fortransmission to the control system, a time stamp being attached to thedisplay content code signal by the clock module.
 24. (canceled) 25.(canceled)
 26. The system according to claim 22 wherein each stimulationdevice comprises with an embedded sensor whose signal is registered by asynchronization device.
 27. The system according to claim 22 furthercomprising a robotic system for driving movements of a limb of the userand configured to provide haptic feedback.
 28. The system according toclaim 22 wherein the clock module is configured to be synchronized withclock module of other systems, including external computers.
 29. Thesystem according to claim 22 further comprising: an exercise logic unitconfigured to generate visual display frames including instructions andchallenges to the display unit; and/or an events manager unit configuredto generate and transmit stimulation parameters to the stimulation unit.30. (canceled)